Pixel multiplication using code spread functions

ABSTRACT

Methods and apparatus for pixel multiplication in optical imaging systems. In one example, an expanded optical point spread function, referred to as a code spread function, is used to phase modulate an incident electromagnetic wavefront, and digital processing, including correlation techniques, are used to filter and process the modulated wavefront to recover sub-pixel information from an image produced by the wavefront on a pixilated detector array.

BACKGROUND

There have been various techniques described and employed to increasethe effective number of pixels delivered by a staring focal plane arraysensor. Some techniques including dithering the image with respect tothe focal plane, or using Hadamard masks in conjunction with the focalplane. These techniques, and others, effectively subdivide the detectorso that it delivers an increased multiplicity of pixels. In most casesthe subdivision provides a factor of four increase in pixel count bysubdividing the detector into a two by two pixel array.

SUMMARY OF INVENTION

Aspects and embodiments are directed to a method of pixel multiplication(e.g., doubling, quadrupling or otherwise increasing the effectivenumber of pixels) in a detector array that involves spatially modulatingthe wavefront in the lens pupil plane. According to one embodiment, themodulation causes the point spread function of the lens to bedeliberately distorted and spread, becoming a complicated yet compactpattern, referred to as a code spread function, which is spread overmany detectors. Sub-detector information may be recovered by correlationfiltering, as discussed in more detail below.

According to one embodiment, a method of pixel multiplication in anoptical imaging system comprises receiving a wavefront ofelectromagnetic radiation at an entrance aperture of the optical imagingsystem, propagating the wavefront to a pupil plane of the opticalimaging system, modulating the wavefront at the pupil plane with amodulation pattern based on a predetermined code spread function for theoptical imaging system to produce a modulated wavefront, propagating themodulated wavefront to an imaging detector which includes an array ofdetector pixels, each pixel having a pixel width, sampling the modulatedwavefront at the imaging detector to produce a sampled data set, anddigitally processing the sampled data set the produce an image. Thedigital processing includes replicating the sampled data set to produceat least two sampled data sets, individually filtering the at least twosampled data sets in parallel with corresponding digital correlationfilters each having a filter function based on the predetermined codespread function to produce at least two filtered data sets, andinterleaving the at least two filtered data sets to produce the image.

In one example propagating the wavefront to the pupil plane of theoptical imaging system includes Fourier transforming the wavefront. Inanother example propagating the modulated wavefront to the imagingdetector includes Fourier transforming the modulated wavefront. Thedigital processing of the sampled data set may further include Fouriertransforming the sampled data set to produce a transformed data set, andreplicating the sampled data ma include replicating the transformed dataset. In one example replicating the transformed data set includesreplicating the transformed data set three times to produce four sampleddata sets, and each digital correlation filter corresponds to a shiftedcode spread function pattern corresponding to a half pixel widthrelative shift of the predetermined code spread function on the imagingdetector. In another example filtering the at least two sampled datasets includes filtering the four sampled data sets by multiplying eachsampled data set by a complex conjugate of the corresponding shiftedcode spread function pattern. In one example the predetermined codespread function has a non-zero average value, and the digital processingfurther includes band pass filtering the image to produce a filteredimage. The digital processing may further include applying a recoveryprocess to the filtered image to recover low spatial frequencyinformation. In one example this recovery processing includes Fouriertransforming the filtered image to produced a transformed image dataset, passing the transformed image data set through a spatial frequencycompensation filter to produce a filtered data set, and Fouriertransforming the filtered data set to recreate the image.

The method may further comprise generating the predetermined code spreadfunction by converting a point object having a predetermined intensityto an amplitude function, propagating the amplitude function to thepupil plane by Fourier transforming the amplitude function, in theFourier domain, multiplying the amplitude function by the modulationpattern to produce a modulated function, propagating the modulatedamplitude function to an image plane of the imaging detector by applyingan inverse Fourier transform to produce a spatially constrainedamplitude pattern, and converting the spatially constrained amplitudepattern to an intensity pattern though multiplication of the spatiallyconstrained amplitude pattern with its complex conjugate to produce thecode spread function. The method may further comprise partitioning theamplitude function in the image plane into two spatially distinctregions. In one example the method further comprises selectivelyactivating an electro-optically active material to apply the modulationpattern to one of the two spatially distinct regions in the image plane.

According to another embodiment an imaging apparatus comprises animaging detector array including a plurality of pixels, each having apixel width, a lens configured to receive an electromagnetic wavefrontfrom a distant scene, a modulation plate positioned at a pupil plane ofthe lens and configured to modulate the wavefront with a modulationpattern based on a predetermined code spread function for the lens toproduce a modulated wavefront, the lens being further configured tofocus the modulated wavefront onto a focal plane of the imaging detectorarray, and the imaging detector array configured to sample the modulatedwavefront to produce a sampled data set, and a digital image processorcoupled to the imaging detector array and configured to digitallyprocess the sampled data set to produce an image of the scene, thedigital processor configured to replicate the sampled data set toproduce at least two sampled data sets, and including at least twodigital correlation filters each having a filter function based on thepredetermined code spread function and configured to filter acorresponding one of the at least two sampled data sets to produce atleast two filtered data sets, wherein the digital image processor isfurther configured to interleave the at least two filtered data sets toproduce the image of the scene.

In one example each digital correlation filter corresponds to a shiftedcode spread function pattern corresponding to a half pixel widthrelative shift of the predetermined code spread function on the imagingdetector array. In another example wherein the predetermined code spreadfunction has a non-zero average value, and the digital image processorfurther includes a band pass filter configured to filter the image toproduce a filtered image. The modulation plate may be a phase modulationplate, for example. In one example the modulation plate is a switchablemodulation plate including an electro-optically active material and apair of optically transparent electrodes positioned on either side ofthe electro-optically active material. In another example the imagingapparatus further comprises a second switchable modulation plate stackedwith the first switchable modulation plate, and a controller coupled tothe first and second switchable modulation plates and configured toalternately switch on the first and second switchable modulation plates.

Still other aspects, embodiments, and advantages of these exemplaryaspects and embodiments, are discussed in detail below. Embodimentsdisclosed herein may be combined with other embodiments in any mannerconsistent with at least one of the principles disclosed herein, andreferences to “an embodiment,” “some embodiments,” “an alternateembodiment,” “various embodiments,” “one embodiment” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described may beincluded in at least one embodiment. The appearances of such termsherein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one embodiment are discussed below withreference to the accompanying figures, which are not intended to bedrawn to scale. The figures are included to provide illustration and afurther understanding of the various aspects and embodiments, and areincorporated in and constitute a part of this specification, but are notintended as a definition of the limits of the invention. In the figures,each identical or nearly identical component that is illustrated invarious figures is represented by a like numeral. For purposes ofclarity, not every component may be labeled in every figure. In thefigures:

FIG. 1 is a flow diagram of one example of a method and apparatus forgenerating a code spread function according to aspects of the invention;

FIG. 2 is a conceptual diagram of one example of a two dimensional arrayof detectors according to aspects of the invention;

FIG. 3A is a representation of adjacent pixels of a detector illuminatedby point light sources;

FIG. 3B is a representation of a pair of adjacent data samples obtainedfrom the illumination by light sources of FIG. 3A;

FIG. 3C is a representation of an output from the detector resultingfrom processing of the data samples of FIG. 3B;

FIG. 3D is a representation of the adjacent illuminations of FIG. 3Aseparated by an artificially created intervening sample, according toaspects of the invention;

FIG. 4A is a representation of one example of a super-sampled codespread function image according to aspects of the invention;

FIG. 4B is a representation of first and second sub-sample sets takenfrom the super-sampled code spread function image of FIG. 4A accordingto aspects of the invention;

FIG. 5 is a flow diagram of one example of an imaging processing methodaccording to aspects of the invention;

FIG. 6A is a representation of one example of a super-sampled codespread function according to aspects of the invention;

FIG. 6B is a representation of the autocorrelation of the code spreadfunction of FIG. 6A;

FIG. 7A is a representation of one example of a data vector which is theequivalent of a pair of illumination points, according to aspects of theinvention;

FIG. 7B is a representation of an image produced by the convolution ofthe code spread function of FIG. 6A with the data vector of FIG. 7A,according to aspects of the invention;

FIG. 7C is a representation of a filtered pattern produced from theimage of FIG. 7B using conventional auto-correlation techniques;

FIG. 7D is a representation of the autocorrelation of the filteredpattern of FIG. 7C;

FIG. 8A is a representation of a first correlation sample set from thecode spread function of FIG. 6A according to aspects of the invention;

FIG. 8B is a representation of a second correlation sample set from thecode spread function of FIG. 6A according to aspects of the invention;

FIG. 9A is a representation of a signal produced by interlacing thefirst and second correlation sample sets of FIGS. 8A and 8B with half adetector width displacement between them, according to aspects of theinvention;

FIG. 9B is a magnified view of a portion of the signal of FIG. 9A;

FIG. 10A is a representation of the auto-correlation of the code spreadfunction of FIG. 6A for the one dimensional case with two closely spacedpoint sources;

FIG. 10B is an enlarged view of a portion of the signal of FIG. 10A;

FIG. 11 is a flow diagram of one example of a digital processing schemefor recovering an image from a convolved data set using a code spreadfunction according to aspects of the invention;

FIG. 12A is a representation of one example of the band pass filtercharacteristics for an example of a digital filter according to aspectsof the invention;

FIG. 12B is a representation of an example of the correspondingreconstruction of a two point image using the filter of FIG. 12A;

FIG. 12C is an enlarged portion of the image of FIG. 12B;

FIG. 13A is a representation of one example of the band pass filtercharacteristics for another example of a digital filter according toaspects of the invention;

FIG. 13B is a representation of an example of the correspondingreconstruction of a two point image using the filter of FIG. 13A;

FIG. 13C is an enlarged portion of the image of FIG. 13B;

FIG. 14A is a representation of one example of the band pass filtercharacteristics for another example of a digital filter according toaspects of the invention;

FIG. 14B is a representation of an example of the correspondingreconstruction of a two point image using the filter of FIG. 14A;

FIG. 14C is an enlarged portion of the image of FIG. 14B;

FIG. 15A is a representation of one example of the band pass filtercharacteristics for another example of a digital filter according toaspects of the invention;

FIG. 15B is a representation of an example of the correspondingreconstruction of a two point image using the filter of FIG. 15A;

FIG. 15C is an enlarged portion of the image of FIG. 15B;

FIG. 16 is a flow diagram of one example of a process for generatingdigital filters in a digital simulation according to aspects of theinvention;

FIG. 17 is a flow diagram corresponding to one example of an opticalsystem for implementing the process of digital correlation filtergeneration of FIG. 16, according to aspects of the invention;

FIG. 18 is a representation of a portion of one example of a code spreadfunction according to aspects of the invention;

FIG. 19A is a representation of the positive amplitude portions of thecode spread function of FIG. 18;

FIG. 19B is a representation of the negative amplitude portions of thecode spread function of FIG. 18;

FIG. 20 is a block diagram of one example of a switchable phasemodulation plate according to aspects of the invention; and

FIG. 21 is a block diagram of another example of a switchable phasemodulation plate according to aspects of the invention.

DETAILED DESCRIPTION

In optical imaging systems, a lens or other foreoptics focuses incomingelectromagnetic radiation from a scene onto an imaging sensor. Thefollowing discussion may refer primarily to a lens as the foreopticselement; however, those skilled in the art, given the benefit of thisdisclosure, will appreciate that any of a wide variety of foreoptics maybe used. The imaging sensor may include a focal plane array sensor orother pixelated array. Conventionally, each detector in the arraycorresponds to one pixel in an image of scene generated by the imagingsensor. Aspects and embodiments are directed to methods of pixelmultiplication that deliberately corrupt the point spread functionassociated with the lens to effectively sub-divide the detectors in thearray and create higher resolution (more pixels) images. Conventionally,the point spread function of the lens is matched to the detector arrayin some way. As discussed in more detail below, according to certainembodiments, the wavefront of the incoming electromagnetic radiation isspatially modulated in the lens pupil plane, thereby causing the pointspread function of the lens to be deliberately distorted and informationfrom the point spread function is distributed over many detectors in thearray. This modified point spread function is referred to herein as acode spread function. Sub-detector information, or information below theresolution of the detector array, may be recovered by correlationfiltering, as discussed further below.

According to certain embodiments, a computational optics device andassociated method includes inserting a phase modulation plate into thepupil plane of the lens to spatially phase modulate the lens pointspread function (PSF), thereby changing it into a much more broadlydistributed patch of point source caused illumination, referred to asthe code spread function (CSF). The intensity distribution within thispatch may be such that it is sharply autocorrelated. Thus, an equivalentPSF may be synthesized through correlation processing. An advantage ofthis approach is that information contained within each detector isredistributed to many surrounding detectors. As a consequence,correlation techniques may be used to recover and resolve imageinformation from regions much smaller than a detector. In some examples,multiple CSF sampled data sets are created by laboratory calibrationwhereby a collimated point source is laterally shifted by a fraction ofa detector width. These data sets are converted into correlationfilters. Subsequent interleaving of the images produces by thesecorrelation filters generates an image with twice the resolution in thex and y directions. As discussed in more detail below, digital filteringtechniques may be used to eliminate the DC pedestal that occurs with anon negative CSF. In other examples optical techniques, using switchedmodulation plates in the lens pupil plane, may provide a zero bias CSF,thereby eliminating the pedestal.

It is to be appreciated that embodiments of the methods and apparatusesdiscussed herein are not limited in application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the accompanying drawings. Themethods and apparatuses are capable of implementation in otherembodiments and of being practiced or of being carried out in variousways. Examples of specific implementations are provided herein forillustrative purposes only and are not intended to be limiting. Also,the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use herein of“including,” “comprising,” “having,” “containing,” “involving,” andvariations thereof is meant to encompass the items listed thereafter andequivalents thereof as well as additional items. References to “or” maybe construed as inclusive so that any terms described using “or” mayindicate any of a single, more than one, and all of the described terms.

The principle of the code spread function discussed herein may bederived by analogy with source compensation holography. Holographicreconstruction may be modeled as a combination of correlation andconvolution within the context of Fourier transforms. For example,consider source information, located in the diffraction far field of thehologram, encoded as an image wave with spatial pattern s(x). Thispattern is defined as being in the source plane. Define a correspondingreference pattern to be r(x) also in the source plane. Both the spatialpattern and the reference pattern may be propagated to the far field(effectively the pupil plane) to create conjugate patterns S(w) andR(w), where S(w) is the Fourier transform of s(x) and R(w) is theFourier transform of r(x). Adding the two transformed (wavefront)patterns together to create an interference pattern, and recording theresulting intensity pattern in a recording medium, such as photographicfilm, produces:H(w)=[S(w)+R(w)][S*(w)+R*(w)]=H(w)=|S(w)|²+|(R(w)|²+S(w)R*(w)+R(w)S*(w)  (1)The term of interest in Equation (1) is:H=S(w)R*(w)  (2)

During reconstruction of the hologram, the reference pattern r(x) isagain propagated through the optical system, and transformed to becomethe reference wave R(w). This reference wave is modulated by thehologram term H given in Equation (2). The result is:R(w)H=R(w)[S(w)R*(w)]=S(w)|R(w)|²  (3)In conventional forms of holography the reference wave is a section of aspherical wave (a plane wave in the limit case). In this case |R(w)|² isconstant and S(w) is reconstructed. S(w) may then be propagated to thefar field to become the reconstructed source pattern s(x). If |R(w)|² isnot constant, but r(x) has the property of having a very shortautocorrelation length, then a good reconstruction of s(x) can still beobtained. This process is referred to as called source compensationholography.

Referring to Equation (3), if a Fourier transform is taken of the middleor right hand term, the result is s(x) convolved with theautocorrelation of r(x). In the source compensation circumstance wherethe autocorrelation of r(x) is point-like, this autocorrelationresembles a conventional optical point spread function, andmathematically is approximated by a Dirac delta function. Convolving thedelta-like function with s(x) allows for recovery of the function s(x).

Source compensation generally involves inserting a spatially correlatedphase modulation plate into a conventional planar or spherical referencewave. The modulation plate may be a sheet of ground glass, or a piece ofshower glass, for example. In most cases the Fourier transform of thespatially modulated reference wave produces an autocorrelation functionin the source plane which resembles a Dirac delta function, andtherefore will give a good quality reconstruction of the source pattern.Those skilled in the art will appreciate, given the benefit of thisdisclosure, that in order for a good reconstruction of the source waveto occur, the spatial modulation pattern must be duplicated exactly.Shifting the position of the reference modulator, R(w), creates anentirely new pattern in the source plane and also a completely differentholographic interference pattern. Thus, good reconstruction of aparticular image may occur only when the reference modulator ispositioned well within its own spatial autocorrelation length.Furthermore, although the central point of the autocorrelated far fieldpattern may be delta function point-like, the sidelobes may be verydifferent. For example, if the modulation is a random phase pattern, thesidelobes of the far field pattern will be a field of randomintensities.

Referring to FIG. 1, there is illustrated a flow diagram correspondingto one example of a method and apparatus for generating a code spreadfunction in accord with one embodiment. In generating a code spreadfunction (CSF) electromagnetic radiation from a landscape source planeis propagated through a lens, and spatially modulated at the lens pupilplane. The modulated electromagnetic radiation then proceeds toilluminate the focal plane. This general flow of electromagneticradiation, from the source to the focal plane, is illustrated by FIG. 1.Electromagnetic radiation is transmitted or reflected by an illuminatedlandscape 110, and the wave of electromagnetic radiation propagates(120) to the pupil plane of the lens 140. During this propagation 120the electromagnetic radiation undergoes diffraction. If the lens 140 issufficiently distant from the observed landscape 110, the diffractionwill be Fraunhofer. A Fraunhofer diffracted wavefront undergoes aFourier transformation. Thus, Fourier transformations are appropriatefor modeling such an optical imaging system.

According to one embodiment, when the wavefront reaches the lens pupilplane it is spatially modulated by a phase plate 130. In one example,the phase modulator 130 is spatially varying in such a way that itchanges the relative phases of each separate portion of the wavefront.The specific spatial modulation pattern is determined by the desiredform of the desired code spread function, as discussed further below. Insome embodiments a phase modulating plate is preferred because phasemodulation does not attenuate the electromagnetic wave. However, inother embodiments modulation plates which are partially absorbing may beused. In certain examples, the spatial modulation of the wavefront bythe phase modulator 130 may be considered analogous to phase platemodulation of the reference wave in source compensation holographybecause the pupil plane is the Fourier transform conjugate of the imageplane (where the detector array is located).

After passing through the phase modulator 130, the wavefront is focusedby the lens 140 such that during propagation 150 it is furtherFraunhofer diffracted until it arrives at the focal plane. According toone example, the image 160 produced at the focal plane is modified bythe phase modulation plate 130 in such a way that the true image isconvolved with the code spread function induced by the phase modulationplate. In one example, the result of the pupil plane spatial phasemodulation is that the image of a point source of electromagneticradiation is spread out into a complicated pattern which covers amoderately large group of detectors in the detector array. The codespread function may be defined as this extended area of illuminationfrom a single point source of electromagnetic radiation in the farfield. Moving the point source of electromagnetic radiation will causethe code spread function to correspondingly shift its position.

Conventionally, an image is the result of the convolution of the lenspoint spread function with the array of light sources. According tocertain embodiments, an image produced by an optical system in which thelens produces a code spread function instead of a point-like pointspread function is the result of convolution of the code spread functionwith the source of electromagnetic radiation. However, the convolutionof the source with the code spread function “scrambles” the imageinformation. Accordingly, correlation techniques may be used to“unscramble” this information and produce the true image, as discussedfurther below. The convolved image is converted by the detectors in thefocal plane detector array into a sampled data set 170. This data setmay be treated as a vector or a matrix for further digital processing.

According to one embodiment, an advantage of generating a spread outcode spread function is that information contained within the confinesof a given detector (i.e., sub-detector information) may be retrieved.This result may be achieved because the code spread function depositsportions of this interior information on the surrounding detectors.Thus, the code spread function may allow the artificial creation of anarray of pixels which is more dense than the physical array ofdetectors. In one embodiment, digital processing of the sampled data set170 is used to extract an image which is higher resolution than thedetector array, without the intervention of the code spread function,would be able to deliver. Thus, effectively, embodiments of the opticalsystem may be configured to produce an image in which multiple pixelsare obtained from each detector in the imaging sensor array.

FIG. 2 illustrates, conceptually, a two-dimensional array 200 ofdetectors 210 in which each detector is sub-divided into four pixels220. The pixels 220 form a super-sampled array. In the exampleillustrated in FIG. 2, the conceptual array 200 includes 25 detectorsarranged in a 5×5 grid. Since each detector 210 is divided into fourpixels 220, the total pixel count is 100 arranged in a 10×10 array. Thisscheme may be expanded to any desired array size, and the array 200 neednot be square. According to one embodiment, it is assumed that thenormal point spread function of the lens (without the pupil plane phasemodulation) is significantly smaller than the detector 210. This isillustrated by circles 230 representing points of electromagneticradiation illuminating adjacent detectors.

Referring FIG. 3A illustrates two adjacent detectors 310 illuminated bytwo separate points of light 320. The result is a pair of adjacent datasamples 330, as illustrated in FIG. 3B. In a conventional system, with anarrow point spread function, these two points will be unresolved. Thepoint illuminations may fall anywhere within the boundary of a detectorand still produce the same output from the detector. Although adjacentdetectors are illuminated, the data samples 330 from these light points320 will be fused together to make a single, somewhat elongated,unresolved object 340, as illustrated in FIG. 3C.

In contrast, using a code spread function according to aspects andembodiments of the present invention may double (or otherwise increase)the sampling density through pixel multiplication, such that the pointilluminations 320 are resolved by the placement of an artificiallycreated intervening sample 350, as illustrated in FIG. 3D. The resolvedoutput is two samples 360 with a gap sample in between. Thus, the twoobjects (point illuminations 320 in the illustrated example) can bedistinguished or resolved from each other and the separation betweenthem may be detected in a measurable way.

According to one embodiment, to resolve the two data points 360 asillustrated in FIG. 3D, a mechanism is used which provides adistinguishable measure for each of the two points. As will beappreciated by those skilled in the art, given the benefit of thisdisclosure, this principle may be extended for multiple resolved datapoints. In one embodiment, the mechanism for resolving the data pointsincludes generating multiple code spread functions which change shape,and partially decorrelate, depending on their relative sampling positionwith respect to the detector array. For example, for a two dimensionaldetector array there are four such code spread functions used. For thefollowing one dimensional illustrative example, there are two such codespread functions used. The one dimensional example is discussed belowwith reference to FIGS. 4A and 4B. In this example, the detector array410 includes a linear array of detectors 420. In each of FIGS. 4A and4B, each vertical stroke 430 represents a sample. Alternate samples arerepresented by dotted lines. In this example, the samples are dividedinto two subsets, referred to as the right subsample set 440 and theleft subsample set 450, as illustrated in FIG. 4B. These two subsets440, 450 of samples contribute to separate code spread functions whichmay be used to generate different correlation masks, as discussedfurther below.

FIG. 4A represents a super-sampled code spread function image. In FIG.4A, the spacing between the vertical strokes 430, or super-sampledistance, corresponds to the width of an unmodified point spreadfunction. Two adjacent vertical strokes correspond to the width of adetector 420. As discussed above, according to certain examples, theunmodified point spread function is substantially narrower than thewidth of a detector. When alternate supersamples are extracted, theresult is two patterns (arrays of subsamples 440, 450), as illustratedin FIG. 4B. These two patterns may be somewhat similar but are notidentical, and may be substantially different.

If the super-sample array (FIG. 4A) is a completely random pattern, itscorrelation length will be established by the width of the unmodifiedlens point spread function. If this pattern is projected onto a detectorarray which matches in spacing the normal lens point spread function,the two patterns 440, 450 of FIG. 4B would be completely uncorrelated(except for a D.C. offset term). However, for examples in which thedetectors are larger than the normal lens point spread function (forexample, twice the size as in the illustrated example), the twosubsample patterns will be partially correlated. As discussed furtherbelow, for this example, the correlation between the left and the rightsubsample patterns is about 50% (provided that the D.C. offset term isfirst subtracted). A 50% correlation dip between adjacent point imageseasily meets the traditional criterion for resolving point targets.According to certain embodiments, methods and apparatus provide for thegeneration of an optical code spread function which is reasonablycompact and which becomes fully decorrelated when it is displaced byhalf the width of a detector. Such an optical pattern makes it possibleto distinguish point illuminations on adjacent detectors with asubstantial signal dip between them, as discussed further below.

The following discussion of various embodiments may refer primarily tothe use of random patterns for the code spread function. Random patterncode spread functions may be advantageous in that the auto-correlationmay produce a single sharp peak with relatively low sidelobes.Periodicity in the code spread function pattern may produce multiplepeaks in the auto-correlation function. However, embodiments are notconstrained to random patterns, and any pattern having a suitably narrowautocorrelation may be used. For example, Hadamard functions may be usedin some embodiments. The pattern, whether random or not, is generated byan appropriate spatial modulator located in the pupil plane of the lensor other foreoptics. In some embodiments, a phase-only modulator ispreferred; however, other embodiments may use modulators which partiallyabsorb the passing light, although there will be light loss with suchmodulators.

Many code spread functions include a D.C. (constant) offset, which mayintroduce a significant disturbance in the image measurement process.Accordingly, techniques to address this “pedestal” problem as discussedfurther below. However, for the purposes of illustration, the followingexamples will assume a code spread function with a zero mean.

One embodiment is directed to a process of resolving two points ofillumination which are spaced a detector width apart. A flow diagramcorresponding to one example of this process is illustrated in FIG. 5.The following examples involve one dimensional experiments; however,those skilled in the art will appreciate, given the benefit of thisdisclosure, that the process may be readily generalized to twodimensions.

In a first step 510, an optical code spread function is generated.Referring to FIG. 6A, there is illustrated one example of asuper-sampled, zero mean code spread function produced by a randomprocess. This code spread function 600 is non-physical in mostcircumstances because a detector cannot measure negative light (exceptwhere a local oscillator wave is present to provide a phase reference).In one embodiment, as illustrated in FIG. 6, the code spread function600 is confined to a limited portion of the detector area or focalplane. According to one embodiment, a guard border is established at theedge of the focal plane, the guard border being half the width of thecode spread function, thus resulting in the confinement of the codespread function. In this border region the resolution of the image maybe corrupted by the correlation process, and may therefore be discarded.Thus, there is a trade between the size of the code spread function andthe area of the image that can be recovered. Increasing the size of thecode spread function increases the signal to sidelobe ratio, but alsodecreases the useful area of the image.

In step 520, the code spread function is propagated to the Fourierdomain and a complex conjugate correlation filter is constructed fromit. In one example, the filter has the property of being a phase-onlyfilter. As a result, the filter modifies the phases of different samplesof the passing signal, but does not change the amplitude. In step 530the autocorrelation of the code spread function is obtained. Theautocorrelation of the code spread function 600 is illustrated in FIG.6B. The autocorrelation of the code spread function 600 has a sharplydefined, and very narrow, peak 610 and a surround of random sidelobes.The autocorrelation is produced by propagating the code spread function600 to the pupil domain through a Fourier transform. The resulting dataset is multiplied with the correlation filter and the product is inverseFourier transformed back to the image domain. FIG. 6B illustrates theresult of this process for a single point object. The same process maybe performed with an extended image data set. During this digitalprocessing of the image, the sharp peak serves essentially the samepurpose as the narrow and peaked point spread function of a conventionallens. Thus, in one example, this peak 610 convolves with the trueimagery to produce the real measured imagery.

According to one embodiment, the sidelobe structures produced byautocorrelation of the code spread function are different from those ofa conventional PSF. In one example, with a random code spread functionthe peak to sidelobe ratio is approximately the square root of thenumber of samples contributing to the code spread function (i.e., thewidth of the code spread function). Thus, by increasing the size of thecode spread function, the peak to sidelobe ratio will also be increased,though not in proportion. As will be appreciated by those skilled in theart, given the benefit of this disclosure, in the case of twodimensional images, the code spread function is also two dimensional. Inthe two dimensional case the peak to side lobe ratio is linearlyproportional to the width of the code spread function. Thus, forexample, a 64 by 64 code spread function will have a peak to side loberatio of 64:1. This provides an image quality which favorably compareswith that produced by a diffraction limited lens with a circular clearaperture.

In one example, the above-discussed approach may be used to spatiallyresolve two points of light which fall on adjacent detectors, asillustrated in FIG. 2, for example. FIG. 7A illustrates an example of adata vector 710 which is the equivalent of a pair of illuminationpoints. The data vector has two non-zero points 720, 725 which are twosupersamples apart. Referring again to FIG. 1, in this exampleFraunhofer diffraction propagation 120, of far field light sources tothe pupil plane of the lens 140, is emulated by a Fast Fourier Transform(FFT) of the input data vector 710. In other words, the input two pointdata vector 710 is Fourier transformed to reach an equivalent of thelens pupil plane. At this Fourier transformed location the data vector710 is multiplied by a phase modulation pattern (130). In one example,the phase modulation pattern is derived from the Fourier transform ofthe desired code spread function, such as that illustrated in FIG. 6A,for example. The further propagation from the lens to the focal plane(150) is emulated by an inverse FFT. In this example, taking the inverseFourier transform of the modulated data vector results in an imageproduced by the convolution of the code spread function 600 with thedata vector 710. This image 730 is shown FIG. 7B. The resulting image730 may be “unscrambled” to recover and resolve the two sample points720, 725 of the original data vector 710 illustrated in FIG. 7A.

Using only conventional auto-correlation techniques, the individualdetectors 210 are too large to properly resolve the structure of thedata vector 710. In effect, the detectors 210 act to low-pass filter thedata vector 710, and convert the image 730 of FIG. 7B into a filteredpattern 740 as shown in FIG. 7C. This pattern is what the detector array“sees” and adjacent image points are combined. Autocorrelation of thefiltered pattern 740 produces a pattern 750 in which the two points 720,725 are unresolved and appear as a single peak 760, as shown in FIG. 7D.

In contrast, aspects and embodiments employ the code spread function toresolve the image points 720, 725 and obtain a higher resolution finalimage. According to one embodiment, the full resolution code spreadfunction is partitioned into two sample data sets, namely a first dataset and a second data set. The second data set corresponds to an imageshift of half a pixel with respect to the first data set. Thispartitioning scheme creates two new code spread functions, one for eachposition of the image, as illustrated in FIGS. 4A and 4B. Each of thesetwo partitioned code spread functions may then be separately correlatedwith the image delivered by the physical detectors. This correlationproduces two sub images, referred to as a left correlation sample setand a right correlation sample set, each corresponding to a slightlydifferent displacement with respect to the detector sampling grid.Examples of the left and right correlation sample sets are illustratedin FIGS. 8A and 8B, respectively. Interlacing the left and rightcorrelation sample sets, with half a detector width displacement betweenthem, produces a new sample set with twice the number of samples andtwice the effective resolution. FIG. 9A illustrates an example of theinterlaced result (corresponding to FIGS. 8A and 8B), with the desiredimprovement in spatial resolution. FIG. 9B illustrates a magnified viewof the now resolved double peak 910.

The example super-sampled code spread function 600 illustrated in FIG.6A includes both positive and negative values. However, in practice, thedetectors 210 of detector array 200 record incident energy in a lightwave, and this energy is never negative. As a result, in the majority ofoptical imaging systems, the pattern projected onto the focal plane ofthe detectors will be positive definite (except in the case of coherentoptical systems which directly detect the amplitude distribution of thewavefront). Accordingly, the code spread function will have a positiveD.C. bias, as discussed above, which leads to the “pedestal problem.” Ifthe code spread function is a uniformly distributed random function thisbias term will be an additive constant within the extent of the codespread function. Such a constant signal with a finite extent is oftencalled a “box” function. Referring to FIGS. 10A and 10B, for a codespread function with a D.C. bias, during the correlation processdiscussed above, two correlations take place simultaneously andlinearly. The first is the autocorrelation of a random-like functionwhich has a zero average value. The second autocorrelation is that ofthe D.C. bias term. This second autocorrelation produces a triangularfunction for one dimensional box functions and a pyramidal function fora two dimensional box function. FIGS. 10A and 10B illustrate an exampleof the one dimensional case with two closely spaced point sources. As isapparent from FIGS. 10A and 10B, the pedestal dominates the correlationrecovery process. A similar situation exists for the two dimensionalcase.

According to certain embodiments, techniques are implemented to addressthis “pedestal problem,” as discussed below. It will further beappreciated by those skilled in the art, given the benefit of thisdisclosure, that certain optical systems may be implemented whicheffectively synthesize a zero average code spread function. For example,such a system may use switchable phase modulation plates in the lenspupil plane, as discussed further below.

The Fourier transform of a box function is a sin(x)/x, or sinc,function. As discussed below, the majority of the energy in thisfunction is at low spatial frequencies, with the peak value, at zerospatial frequency, being the integral of the D.C. offset of the boxfunction. However, a sinc function has oscillating sidelobes whichextend, at low amplitude, to high spatial frequencies. These highspatial frequency sidelobes are the result of the sharp corners of thebox function. By smoothing these edges of the box the high spatialfrequencies of this function may be suppressed. In one example, this maybe accomplished by smoothly reducing the values of the code spreadfunction values at its edges.

According to one embodiment, digital linear filtering in the Fourierdomain is used to reduce, or preferably eliminate, the effects of thepedestal and recover a high quality image from the convolved data set.There are several different filtering techniques which may beimplemented, as discussed further below. A flow diagram of a generalapproach for digital processing of the convolved data set is illustratedin FIG. 11. According to one embodiment, examples of this processingtechnique is linear and involve only a single pass, in substantialcontrast to most conventional computational imaging techniques whichrequire substantial iteration, and therefore computation, to recover ahigh quality image. In certain aspects the digital processing of FIG. 11emulates the optical flow discussed above with reference to FIG. 1;however, activity in the digital equivalent of the pupil plane issubstantially different from the optical case.

Referring to FIG. 11, the digital processing begins with the sampledimage 1110. In one example this sampled image 1110 corresponds to thesampled data set 170 of FIG. 1. In step 1115 the sampled image/data set1110 is Fourier transformed to produce a transformed data set. TheFourier transform corresponds to Fraunhofer diffraction propagation fromthe image plane to the lens pupil plane. Thus, the transformed sampledata set has a one-to-one correspondence with the light wavefrontimmediately after the wavefront has passed through the phase modulationplate 130. A function of the digital processing is to distinguish eachof the pixel zones 220 of each of the subdivided detectors 210. This isaccomplished by replicating the transformed data set. The replicatedcopies of the transformed data set are then passed in parallel, throughseparate correlation filters 1125. In the illustrated examples, eachdetector 210 is subdivided into four pixel zones 220, and accordingly,the transformed data set is replicated three times and passed throughfour separate correlation filters 1125. Each filter corresponds to ahalf detector relative image shift in the x and y directions. However,those skilled in the art will appreciate, given the benefit of thisdisclosure, that the general concept is not limited to subdividing eachdetector 210 into four pixel zones 220. In practice, the subdividion maybe conditioned on the relative size of the lens point spread functionand the detector 210. In certain examples, if the lens point spreadfunction is substantially smaller than the detector 210, the system maybe configured to extract more than four pixels for a given detector. Inthis case, more than four correlation filters 1125 would be used. It isfurther to be appreciated that in examples in which the width of thepoint spread function is exactly matched to the detector width (so thatthe maximum signal to noise ratio is obtained), the digital processingdiscussed herein may not improve the spatial resolution, but may providea Nyquist sampled image and therefore eliminate aliasing artifacts.

Still referring to FIG. 11, in one embodiment each correlation filtermultiplies its copy of the transformed data set by the complex conjugateof the shifted code spread function pattern created during sensorcalibration, as discussed further below. Because the optical code spreadfunction pattern has structural details which are substantially finerthan the size of the detector 210, each of these four filters mayextract information from a different portion of each detector. As aresult, a higher resolution final image may be obtained. In addition tocompensating for the phase modulation applied by the phase modulator 130of FIG. 1, the digital correlation filters 1125 may also be configuredto reduce the effect of the D.C. offset in the code spread function toan acceptable level, as discussed further below.

After the copies of the transformed data set are correlation filtered(by filters 1125), each set is separately Fourier transformed (step1135) back to the image plane. The result is four separate sub-images1120 which have been reconstructed with point spread functions producedby autocorrelation. These sub-images 1120 contain the high spatialfrequency information used to resolve objects at the pixel spacinglimit. In one example each of these sub-images 1120 will be displaced byhalf a detector width with respect to each other. These sub-images 120are then interleaved in step 1145 to create a resulting image 1130 whichhas four times as many pixels as the focal plane detector array canproduce by itself.

According to certain embodiments, additional processing steps 1155,1165, and 1175 may be added to fully recover a high quality image 1150.The digital correlation filters 1125 may also band pass filter (usuallylow pass filter) the imagery to reduce, or eliminate, the triangularD.C. pedestal discussed above. The pedestal generally involves lowspatial frequency information whereas the high resolution (pixelmultiplied) information has high spatial frequency. Accordingly, afterreassembly of the fine details of the image, the low spatial frequencyinformation may be restored to avoid distortion of the final image 1150.

Still referring to FIG. 11, interleaving the sub-images 1120 at step1145 produces a single image 1130 with more effective pixels than thenumber of detectors 210 in the focal plane array (FPA) 200. Thisexpanded image 1130 resolves fine structure that otherwise would not bevisible in a conventional sensor. This image 1130 also has reducedcontributions from the filtered spatial frequency components of theoriginal source image, as discussed above. Thus, in order to reestablishthis low spatial frequency information, the bandpass filtering in thedigital processing may be inverted in the following recovery procedure.

In one embodiment, the recovery procedure begins with delivery of theinterleaved imagery (image 1130) to a Fourier transform device where itis transformed (step 1155) back into the spatial frequency domain (i.e.the equivalent of the pupil plane). In step 1165, the resulting data setis passed through a spatial frequency compensation filter, which undoesthe bandpass filtering described above. This background filter may bedesigned to pass only the pedestal portions of the spectrum. In anotherexample, the background filter may undo the bandpass filtering from thefilters 1125 and also pass the high spatial frequencies. The reversefiltered result undergoes a second Fourier transform at step 1175 whichrecreates the image 1140. In examples where the background filter atstep 1165 corrects for the D.C. pedestal, and also passes the highspatial frequency information, the reconstructed image 1140 is thefinished product (equivalent to final image 1150). In other examples,where only the pedestal information is reconstructed in steps 1155-1175,the reconstructed image 1140 is combined with the interleaved highspatial frequency imagery (from 1130) to produce the composite finalimage 1150.

As discussed above, the digital correlation filters 1125 may beconfigured to reduce or eliminate the effects of the pedestal in thecode spread function. There are several different bandpass filterprofiles which may be implemented, some examples of which are discussedfurther below. In each of these examples, the filter effectivelysuppresses the pedestal, and also substantially reduces the low spatialfrequencies in the imagery and thereby causes distortion of the image.However, as discussed above, by placing a constraint on these filtersthe low spatial frequency information can be recovered and the imagedistortion eliminated. This constraint is that the filters do not reducethe amplitudes of the low frequencies to zero, such that the amplitudesmay be restored to their original values using embodiments of therecovery process discussed above. The low spatial frequencies gathertheir energy from a wide area (many detectors). In contrast, fine detailhigh spatial frequency information is highly localized. Thus, the signalto noise ratio for the low spatial frequencies is much higher than thatof the high spatial frequencies. This high signal to noise ratio allowsrestoration of the low spatial frequency information by inversion of thebandpass filter, as discussed above.

According to one embodiment, the correlation filters 1125 are configuredsuch that all parts of the Fourier domain sinc function that are above agiven threshold are proportionately reduced in value. This techniquereduces the intensity of the image in the Fourier domain at areascorresponding to the D.C. pyramid. An example of this technique isillustrated in FIGS. 12A-C. FIG. 12A illustrates an example of the bandpass filter characteristics. FIG. 12B illustrates an example of thecorresponding reconstruction of a two point image. FIG. 12C is anenlarged portion of the image of FIG. 12B showing the two points (onadjacent detectors) fully resolved. Restoration of the filtered lowspatial frequency information may be achieved through inversion of thefilter function of FIG. 12A after the separate sub-images 1120 have beeninterleaved (step 1145), as discussed above. This filtering techniqueleaves a very small pedestal and very cleanly resolves the two adjacentsignals, as demonstrated in FIG. 12C. As will be appreciated by thoseskilled in the art, given the benefit of this disclosure, thecorrelation filter design includes adjustable parameters. The exampleillustrated in FIGS. 12A-C is not necessarily optimized in terms of theeffects of adjusting these parameters. Accordingly, this example isillustrative only, and not intended to be definitive. Parameteroptimization may improve the images delivered by the correlation filtersand digital process.

According to another embodiment, the correlation filters 1125 areconfigured to set all of the parts of the image and filter spectra tounity magnitude. This filtering technique leaves a flat spectrum afterfiltering, as shown in FIG. 13A. In other words, the filtering techniquedelivers a phase-only Fourier spectrum. Without low frequencycompensation, the reconstructed image strongly emphasizes the highspatial frequency content of the imagery. Strictly speaking the constantspectrum amplitude of FIG. 13A is produced by non linear filtering sinceboth the correlation filters 1125 and the image data set 1110 areadjusted, frequency by frequency, to produce a flat output spectrum.However, the digital processing remains single pass, with no processingiteration required. FIG. 13A illustrates an example of the correspondingreconstruction of a two point image. FIG. 13C is an enlarged portion ofthe image of FIG. 13B showing the two points (on adjacent detectors)fully resolved.

According to another embodiment, an inverted triangle low pass filter isused to reduce the low spatial frequencies. An example of this techniqueis illustrated in FIGS. 14A-C. FIG. 14A illustrates an example of thefilter profile. The sharp corners of this filter profile may generatesome high frequency ringing. FIG. 14B illustrates an example of thecorresponding reconstruction of a two point image. As may be seen withreference to FIG. 14B, in this example, the pedestal is not completelyeliminated, but is greatly reduced. As will be appreciated by thoseskilled in the art, given the benefit of this disclosure, thiscorrelation filter design includes adjustable parameters. The exampleillustrated in FIGS. 14A-C is not necessarily optimized in terms of theeffects of adjusting these parameters. Accordingly, this example isillustrative only, and not intended to be definitive. Parameteroptimization may improve the images delivered by the correlation filtersand digital process.

According to another embodiment, the correlation filters 1125 may beconfigured with a profile that is an inverted Gaussian function. Anexample of this technique is illustrated in FIGS. 15A-C. FIG. 15Aillustrates an example of the filter profile. The smoothedcharacteristic of a Gaussian function may reduce the high spatialfrequency ringing. FIG. 15B illustrates an example of the correspondingreconstruction of a two point image. As may be seen with reference toFIG. 14B, in this example, the pedestal is not completely eliminated,but is greatly reduced. As will be appreciated by those skilled in theart, given the benefit of this disclosure, this correlation filterdesign includes adjustable parameters. The example illustrated in FIGS.15A-C is not necessarily optimized in terms of the effects of adjustingthese parameters. Accordingly, this example is illustrative only, andnot intended to be definitive. Parameter optimization may improve theimages delivered by the correlation filters and digital process.

The above-discussed principles and examples may be demonstrated bydigital simulation. In addition, as discussed above, an operationsoptical/digital sensor system may include an ensemble of digitalfilters. In both operation and simulation, image reconstruction may beperformed with digital processing, as discussed above. According to oneembodiment, the code spread function is specified analytically and aphase modulation plate is derived from the analytical code spreadfunction.

Referring to FIG. 16 there is illustrated one example of a flow diagramfor creation of digital filters in a digital simulation. In FIG. 16, FFrefers to the far field; PP refers to the pupil plane (at lens 140),which corresponds to the spatial frequency domain; IP refers to theimage plane (at the focal plane); and FT is an abbreviation for FourierTransform. The process begins with creation of a patch 1610 in the imageplane (IP). The contents of this patch 1610 may be some complicatedfunction which has a very narrow autocorrelation function. The size ofthe patch 1610 is determined by a requirement for the signal to sideloberatio and by the need to minimize the exclusion zone at the edges of theimage. In some examples the patch 1610 is square, however, it need notbe. In one example the patch 1610 is filled with uniform distributionrandom values. In other examples the patch 1610 may include a soft edgewindowed function, randomly positioned spikes of unity value or anyother complicated function which is narrowly autocorrelated. In certainexamples Hadamard functions may be used for the patch 1610.

At step 1620 the patch 1610 is Fourier transformed to create aconstrained spatial frequency spectrum function. One example of acompact patch 1610 is a windowed function. The Fourier transform of sucha patch is a convolution of the Fourier transformed window with theFourier transform of the example pseudorandom process. This convolutiontypically smoothes the Fourier spectrum and leads to wider-spacedoscillations of the Fourier components.

At stage 1630, in the pupil plane (pp), the spectrum representation ofthe initial image patch may serve as the master pattern for producing awavefront modulation plate. The spectrum, as transformed from theinitial image patch 1610, may have graded amplitude values as well asphase variations. As discussed above, in many cases it may be desirableto have a phase-only modulation plate. Thus, only the phase informationin the initial spectrum obtained at step 1620 may be used. One exampleis a two state phase modulation plate. Such a phase modulation patterncan be derived from the initial image spectrum by splitting the phaseregion into two complimentary parts and assigning zero phase change toone of these two phase regions and a 180° phase change to the otherhalf. This phase pattern may then be used to define the phase modulationplate 1640.

According to one embodiment, the code spread function may be defined ina multi-step process 1650. In one example this process 1650 begins witha single point object 1652 in the far field source plane, of someintensity. This object 1652 is converted to an amplitude 1654 by takingthe square root of the intensity. If the intensity is initially of unitstrength, the amplitude will likewise be of unit value. The resultingamplitude spike is propagated to the pupil plane by means of a Fouriertransform 1656. In the Fourier domain the amplitude is multiplied by themodulation function of the modulation plate 1640. The resultingmodulated amplitude function may be propagated to the image plane byapplying an inverse Fourier transform at step 1660. The result of theinverse Fourier transform is a spatially constrained amplitude patternof complicated and complex form. This amplitude pattern may be convertedto an intensity pattern at step 1670 through multiplication with itscomplex conjugate.

Still referring to FIG. 16, the intensity pattern resulting from step1670 may have structural variations which are substantially finer thanthe detector array sample spacing. In one embodiment, from this masterintensity pattern a set of displaced and condensed code spread functions1680 can be extracted. In one example, in which each detector issub-divided into four sub-pixels, as discussed above, four such codespread functions may be extracted, as shown in FIG. 16. In this case,each “sub-code spread function” corresponds to one of four possiblepositions of a point object on a detector. Each sub-code spread functionmay be created by a process of pixel accumulation. For example, fourdisplaced pixels additively contribute to each sub-code spread function.As discussed above, in one example these contributions are from fourrelatively displaced positions; the displacement distance being onepixel. Table 1 below provides an example of a mechanism by which tocalculate the displacements.

TABLE 1 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 4344 45 46 51 52 53 54 55 56 61 62 63 64 65 66

Table 1 illustrates a conceptual 3×3 array of detectors, each detectorsub-divided into four regions, resulting in a 6×6 array of subpixels.Each conceptual subpixel represents a sample in the two dimensionalsample set delivered by the transformative processes leading to step1670 in FIG. 16. According to one embodiment, the four sub-code spreadfunctions may be created as a generalization of the following scheme:

Referring to Table 1, to produce the first sub-code spread function1681, add subpixels 11, 12, 21, 22 to produce the first combined samplein the first column, first row. Similarly, add pixels 13, 14, 23, 24 forthe second sample in the second column, first row. Add subpixels 31, 32,41, 42 for the adjacent sample in the first column, second row, etc.This results in the combined subpixel set illustrated by Table 2:

TABLE 2 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 4344 45 46 51 52 53 54 55 56 61 62 63 64 65 66

To produce the second sub-code spread function 1682, add pixels 12, 13,22, 23; 14, 15, 24, 25; 32, 33, 42, 43; etc. This produces a displacedsubpixel combination shown in Table 3:

TABLE 3 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 4344 45 46 51 52 53 54 55 56 61 62 63 64 65 66

To produce the third sub-code spread function 1683, add subpixels 21,22, 31, 32; 23, 24, 33, 34; 41, 42, 51, 52; etc. This combination isshown in Table 4:

TABLE 4 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 4344 45 46 51 52 53 54 55 56 61 62 63 64 65 66

To produce the fourth sub-code spread function 1684, add subpixels 22,23, 32 33; 24, 25, 34, 35; 42, 43, 52, 53; etc. This is illustrated byTable 5:

TABLE 5 11 12 13 14 15 16 21 22 23 24 25 26 31 32 33 34 35 36 41 42 4344 45 46 51 52 53 54 55 56 61 62 63 64 65 66

Thus, the sub-code spread functions may be generated from the masterimage plane, high resolution code spread function obtained at step 1670.As discussed above, from the code spread functions, correlation masks,or digital filters, may be created which operate in the digital Fourierdomain. To produce these filters, each of the sub-code spread functionsis Fourier transformed at step 1690. At step 1695, the complexconjugates of the spatial frequency spectra produced at step 1690 aregenerated to create the corresponding correlation mask patterns.

Light propagates as an amplitude, but is detected as an intensity.Fourier transforms may operate on either an amplitude function or anintensity function. It is to be appreciated that to properly simulate apixel multiplication system and process as discussed above, the Fouriertransforms operate on amplitudes when simulating the lens and onintensity when simulating the far field object and the focal planeimage. If these transforms are operated correctly, the signal tosidelobe ratio is, to good approximation, the square root of the numberof samples in the code spread function patch, whereas the signal tosidelobe ratio may be significantly worse if the transforms are notoperated correctly.

According to one embodiment, creation of a physical pupil planemodulation plate and digital correlation filters involves both opticalmeasurements and digital processing. FIG. 17 is a flow diagramillustrating an example of an optical system for producing theabove-discussed digital correlation filters. In one embodiment, in orderto create the sub-code spread function correlation filters (1695), anoptical pupil plane phase modulation plate may be produced first. Theprocess for determining the phase pattern of this plate is describedabove with reference to FIG. 16. After the phase modulation plate 130has been fabricated, and inserted in the optical system in associationwith the lens 140, the lens system may be optically calibrated and thedigital correlation filter masks may be created.

Referring to FIG. 17, in one embodiment the process of creating thedigital correlation filter masks may begin by generating a far fieldpoint of light (step 1710). This may be done with a bench collimator. Ina point measurement mode, a collimator may emit a plane wave whichpropagates to the lens (step 1720). In the pupil plane of the lens thewavefront passes through the modulation plane 130, and its spatial phasepattern is appropriately modified. The wavefront further propagates tofocus on the focal plane (1730). At the focal plane of the opticalsystem is located an array of detectors which converts the intensity ofthe light into a sampled data set (step 1740), as discussed above.Multiple data sets may be generated depending on the relative positionof the far field point of light in step 1710. Each data set may begenerated with the light in a different relative position (step 1750).In one example, the point of light is placed in a position whichcorresponds to one of the subdivided (sub)pixels of each detector. Inthe above-discussed example, this positioning produces four positions ofthe point of the light, with half a detector displacement in the x and ydirections, resulting in four data sets.

According to one embodiment, a digital correlation filter is generatedfor each position of the far field point of light. In one example, theprocess of filter, or mask, generation is as described above withreference to FIG. 16. Referring to FIG. 17, in step 1690, the data setproduced at step 1740 is Fourier transformed to produce spatialfrequency spectrum which is in the equivalent of the lens pupil plane.The spectrum undergoes complex conjugation. The amplitudes of theconjugated spectrum may then be adjusted to reduce, or eliminate, thoseportions of the spectrum which correspond to the D.C. offset pedestal,as discussed above. The result is a pattern, or mask, 1695 which, whenmultiplied by the spectrum from code spread function modified imagery,may allow reconstruction of the image information contained in one ofthe pixels in the subdivided detector. Subsequent interleaving of thesevarious sub-detector pixel sets may create a higher resolution imagethan the original detector array could supply.

As discussed above, an optical system may be implemented whicheffectively synthesizes a zero average code spread function, therebyavoiding the “pedestal problem.” Conventional optical systems projectthe wavefront amplitude of far field objects onto the focal plane. Asnoted above, most detection devices measure only the energy, orintensity, of the wavefront, and therefore negative amplitudes aresquared by the detector to become positive intensities. This results ina non-zero average function and the D.C. bias or “pedestal problem”discussed above. However, according to certain aspects and embodiments,a device which allows detection of the negative amplitude parts of thewavefront may be implemented, which allows for development of a zeroaverage code spread function. The problem of D.C. bias, and itsconcomitant pedestal, is thereby eliminated.

Referring to FIG. 18, it can be seen that the negative portions of theamplitude of the image of the code spread function are located indifferent spatial locations than are the positive portions of theamplitude. Accordingly, the two different spatial regions may beseparately measured. In one example, the code spread amplitude functionof FIG. 18 may be partitioned into two spatially distinct regions, asshown in FIGS. 19A and 19B. Thus, separate correlation filters may beconstructed for the positive and for the negative regions of the highresolution code spread function. In one example, the code spreadfunction configuration for quadrupling the number of pixels of adetector array may thus use eight, rather than four, digital correlationfilters since for each of the four relative filter displacements therewill be separate correlation filters for the positive region and for thenegative region.

According to one embodiment, partitioning of the code spread functioninto different spatial regions may be implemented using switchablemodulation plates. It may be preferable that the switching mechanism iselectrically controlled and relatively fast. In one example low voltageswitching employs a liquid crystal as the optical active substance.However, other technologies, such as ferroelectric andmicro-electro-mechanical (MEMS) systems may be employed. Technology forelectrical switching of spatially patterned phase modulation is welldeveloped for the visible and near infrared portions of theelectromagnetic spectrum. Certain liquid crystals may be used to producea midwave infrared switchable phase modulator. Phase modulation in thelong wave infrared spectrum (e.g., 8 to 12 microns) may rely onferroelectric or MEMS devices.

Referring to FIG. 20 there is illustrated a block diagram of one exampleof a switchable phase modulation plate according to one embodiment. Inthe illustrated example, the phase modulation plate includes anelectro-optically active material 2010 which is placed between twotransparent electrodes 2020 and 2030. At least one of the top electrode2020 and bottom electrode 2030 is spatially patterned so that, when avoltage is applied between the electrodes, the desired spatial patternof phase modulation is impressed on a light wave that passes through themodulation plate. Thus, incoming light 2040 is spatially modulated toproduce two-state, phase modulated outgoing light 2050.

According to one embodiment, the switchable phase modulation plate ofFIG. 20 may be used to alternately produce images with a code spreadfunction or with a conventional lens point spread function. In certainexamples, this selectibility provides the system with an ability todeliver both unmodified images and images which have twice the spatialresolution, but which accentuate the high spatial frequencies relativeto the low spatial frequencies. Combination of these two images mayproduce a balanced, doubled-resolution image.

According to another embodiment, stacking two of the switchablemodulation plates permits the spatial separation of the positive andnegative portions of the code spread function, as discussed above. Anexample of a configuration of a double stacked modulation plate systemis illustrated in FIG. 21. In the illustrated example, the systemincludes a first switchable phase modulation plate 2110 and a secondswitchable phase modulation plate 2120. Each plate 2110 and 2120 mayswitch a different spatial region, for example, corresponding to thepositive and negative amplitude regions of the code spread function, asdiscussed above. In one example, only one of the two modulators 2110 and2120 is switched on at any given time. In this example, operation of thesystem produces a three image sequence. Thus, the incoming light 2040 isspatially modulated by the switchable phase modulation plates 2110, 2120to produce three-state, phase modulated outgoing light 2130.

For example, a first image is generated with both plates 2110 and 2120switched off. This mode of operation produces the first image at thespatial sampling density of the detector spacings in the focal planearray (FPA). A second image may be generated with the positive codespread function region modulation plate 2110 switched on (while thesecond modulation plate 2120 remains off). This mode of operationgenerates the second image corresponding to a partial code spreadfunction for the positive portions of the total code spread function.The high resolution positive region information may be extracted usingfour correlation filters as described above. A third image may begenerated with the positive modulation plate 2110 switched off and thenegative region modulation plate 2120 switched on. The negative codespread function region information may be extracted by four correlationfilters which are dedicated to the negative information, as discussedabove. Subsequently, the two (second and third) high resolutiondecorrelated images (positive and negative) may be reassembled andcombined with the first image to produce a high quality, highresolution, finished image.

It will be appreciated by those skilled in the art, given the benefit ofthis disclosure, that the digital processing discussed herein is linearin nature. Thus, according to certain embodiments, the recorded, codespread function convolved, imagery may only require one pass through thedigital processing to recover the doubled (or otherwise magnified) imageresolution. This linearity is in striking contrast to conventionalcomputational imaging techniques which generally involve many iterationsto recover a high resolution image. Thus, embodiments of the techniquesdiscussed herein may offer a significant reduction in computationalburden.

It is further to be appreciated that the digital processing may beminimized if the code spread function is the same all over the image.For this to be the case, the lens 140 must be isoplanatic andrectilinear. An isoplanatic lens has a point spread function which isthe same everywhere in the field of view of the lens. Rectilinearitymeans that the shape of viewed objects is independent of where theobjects are located in the field of view of the lens. Another propertyof rectilinearity is that the lens does not have barrel, pincushion, orother geometric distortions. However, it is to be appreciated that thetechniques discussed herein do not require the lens 140 to beisoplanatic or rectilinear. It may be difficult to achieve a wide fieldof view with a lens 140 that is both isoplanatic and rectilinear.Accordingly, in certain examples, the pixel multiplication techniquesdiscussed herein may be applied to an image that is locally isoplanaticand rectilinear. In such examples, the image may be made rectilinear bydigital manipulation and interpolation. In these examples, the codespread function may vary from region to region; however, itscharacteristics are measurable in each region. Digital processing may beapplied on a regional basis with local code spread functions for eachregion.

Thus, aspects and embodiments provide processes and systems forimplementing pixel multiplication to produce high resolution imageryusing single pass correlation techniques. As discussed above, a phasemodulation plate may be inserted into the pupil plane of a lens in anoptical imaging system. The spatial modulation alters the point spreadfunction of the lens into a more broadly distributed “patch” of pointsource-originating illumination, referred to herein as a code spreadfunction. The intensity distribution within the code spread function issharply auto-correlated. Multiple code spread function sampled data setsmay be produced through a calibration process wherein a collimated pointsource is laterally shifted by a fraction of a detector width, asdiscussed above. These data sets may be used to generate digitalcorrelation filters. Subsequent interleaving of the images produced bythese digital correlation filters may result in an image that hasincreased resolution (e.g., doubled resolution for the case in whicheach detector is sub-divided into four regions) in the x and ydimensions. In addition, digital filtering provides a mechanism forreducing, or preferably eliminating, the D.C. pedestal that occurs withcode spread functions that have non-zero averages. Furthermore, asdiscussed above, optical techniques, such as the use of a switchablephase modulation plate, may be used to provide a zero-bias code spreadfunction and thereby avoid the pedestal problem.

Having described above several aspects of at least one embodiment, it isto be appreciated various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be part of thisdisclosure and are intended to be within the scope of the invention.Accordingly, the foregoing description and drawings are by way ofexample only, and the scope of the invention should be determined fromproper construction of the appended claims, and their equivalents.

What is claimed is:
 1. A method of pixel multiplication in an opticalimaging system comprising: receiving a wavefront of electromagneticradiation at an entrance aperture of the optical imaging system;propagating the wavefront to a pupil plane of the optical imagingsystem; modulating the wavefront at the pupil plane with a modulationpattern based on a predetermined code spread function for the opticalimaging system to produce a modulated wavefront; propagating themodulated wavefront to an imaging detector which includes an array ofdetector pixels, each pixel having a pixel width; sampling the modulatedwavefront at the imaging detector to produce a sampled data set; anddigitally processing the sampled data set the produce an image, thedigital processing including: replicating the sampled data set toproduce at least four sampled data sets; individually filtering the atleast four sampled data sets in parallel with corresponding digitalcorrelation filters each having a filter function based on a shiftedcode spread function pattern corresponding to a half pixel widthrelative shift of the predetermined code spread function on the imagingdetector to produce at least four filtered data sets; and interleavingthe at least four filtered data sets to produce the image.
 2. The methodof claim 1, wherein propagating the wavefront to the pupil plane of theoptical imaging system includes Fourier transforming the wavefront. 3.The method of claim 2, wherein propagating the modulated wavefront tothe imaging detector includes Fourier transforming the modulatedwavefront.
 4. The method of claim 1, wherein digital processing thesampled data set further includes Fourier transforming the sampled dataset to produce a transformed data set; and wherein replicating thesampled data includes replicating the transformed data set.
 5. Themethod of claim 4, wherein filtering the at least four sampled data setsincludes multiplying each sampled data set by a complex conjugate of thecorresponding shifted code spread function pattern.
 6. The method ofclaim 4, wherein the predetermined code spread function has a non-zeroaverage value, and wherein the digital processing further includes bandpass filtering the image to produce a filtered image.
 7. The method ofclaim 6, wherein the digital processing further includes applying arecovery process to the filtered image to recover low spatial frequencyinformation, the recovery processing including: Fourier transforming thefiltered image to produced a transformed image data set; passing thetransformed image data set through a spatial frequency compensationfilter to produce a filtered data set; and Fourier transforming thefiltered data set to recreate the image.
 8. The method of claim 1,further comprising generating the predetermined code spread function by:converting a point object having a predetermined intensity to anamplitude function; propagating the amplitude function to the pupilplane by Fourier transforming the amplitude function; in the Fourierdomain, multiplying the amplitude function by the modulation pattern toproduce a modulated function; propagating the modulated amplitudefunction to an image plane of the imaging detector by applying aninverse Fourier transform to produce a spatially constrained amplitudepattern; and converting the spatially constrained amplitude pattern toan intensity pattern though multiplication of the spatially constrainedamplitude pattern with its complex conjugate to produce the code spreadfunction.
 9. The method of claim 8, further comprising partitioning theamplitude function in the image plane into two spatially distinctregions.
 10. The method of claim 9, further comprising selectivelyactivating an electro-optically active material to apply the modulationpattern to one of the two spatially distinct regions in the image plane.11. An imaging apparatus comprising: an imaging detector array includinga plurality of pixels, each having a pixel width; a lens configured toreceive an electromagnetic wavefront from a distant scene; a modulationplate positioned at a pupil plane of the lens and configured to modulatethe wavefront with a modulation pattern based on a predetermined codespread function for the lens to produce a modulated wavefront, the lensbeing further configured to focus the modulated wavefront onto a focalplane of the imaging detector array, and the imaging detector arrayconfigured to sample the modulated wavefront to produce a sampled dataset; and a digital image processor coupled to the imaging detector arrayand configured to digitally process the sampled data set to produce animage of the scene, the digital processor configured to replicate thesampled data set to produce at least two sampled data sets, andincluding at least two digital correlation filters each having a filterfunction based on a shifted code spread function pattern correspondingto a half pixel width relative shift of the predetermined code spreadfunction on the imaging detector array and configured to filter acorresponding one of the at least two sampled data sets to produce atleast two filtered data sets, wherein the digital image processor isfurther configured to interleave the at least two filtered data sets toproduce the image of the scene.
 12. The imaging apparatus of claim 11,wherein the predetermined code spread function has a non-zero averagevalue, and wherein the digital image processor further includes a bandpass filter configured to filter the image to produce a filtered image.13. The imaging apparatus of claim 11, wherein the modulation plate is aphase modulation plate.
 14. The imaging apparatus of claim 11, whereinthe modulation plate is a switchable modulation plate including anelectro-optically active material and a pair of optically transparentelectrodes positioned on either side of the electro-optically activematerial.
 15. The imaging apparatus of claim 14, wherein the switchablemodulation plate is a first switchable modulation plate, and furthercomprising: a second switchable modulation plate stacked with the firstswitchable modulation plate; and a controller coupled to the first andsecond switchable modulation plates and configured to alternately switchon the first and second switchable modulation plates.
 16. An imagingapparatus comprising: an imaging detector array including a plurality ofpixels, each having a pixel width; a lens configured to receive anelectromagnetic wavefront from a distant scene; a switchable modulationplate including an electro-optically active material and a pair ofoptically transparent electrodes positioned on either side of theelectro-optically active material, the switchable modulation platepositioned at a pupil plane of the lens and configured to modulate thewavefront with a modulation pattern based on a predetermined code spreadfunction for the lens to produce a modulated wavefront, the lens beingfurther configured to focus the modulated wavefront onto a focal planeof the imaging detector array, and the imaging detector array configuredto sample the modulated wavefront to produce a sampled data set; and adigital image processor coupled to the imaging detector array andconfigured to digitally process the sampled data set to produce an imageof the scene, the digital processor configured to replicate the sampleddata set to produce at least two sampled data sets, and including atleast two digital correlation filters each having a filter functionbased on the predetermined code spread function and configured to filtera corresponding one of the at least two sampled data sets to produce atleast two filtered data sets, wherein the digital image processor isfurther configured to interleave the at least two filtered data sets toproduce the image of the scene.
 17. The imaging apparatus of claim 16,wherein the switchable modulation plate is a first switchable modulationplate, and further comprising: a second switchable modulation platestacked with the first switchable modulation plate; and a controllercoupled to the first and second switchable modulation plates andconfigured to alternately switch on the first and second switchablemodulation plates.
 18. A method of pixel multiplication in an opticalimaging system comprising: generating a predetermined code spreadfunction by: converting a point object having a predetermined intensityto an amplitude function; propagating the amplitude function to a pupilplane by Fourier transforming the amplitude function; multiplying theamplitude function by a modulation pattern to produce a modulatedfunction in the Fourier domain; propagating the modulated amplitudefunction to an image plane of an imaging detector by applying an inverseFourier transform to produce a spatially constrained amplitude pattern;and converting the spatially constrained amplitude pattern to anintensity pattern though multiplication of the spatially constrainedamplitude pattern with its complex conjugate to produce the code spreadfunction; receiving a wavefront of electromagnetic radiation at anentrance aperture of the optical imaging system; propagating thewavefront to a pupil plane of the optical imaging system; modulating thewavefront at the pupil plane with the modulation pattern based on thepredetermined code spread function for the optical imaging system toproduce a modulated wavefront; propagating the modulated wavefront to animaging detector which includes an array of detector pixels, each pixelhaving a pixel width; sampling the modulated wavefront at the imagingdetector to produce a sampled data set; and digitally processing thesampled data set the produce an image, the digital processing including:replicating the sampled data set to produce at least two sampled datasets; individually filtering the at least two sampled data sets inparallel with corresponding digital correlation filters each having afilter function based on the predetermined code spread function toproduce at least two filtered data sets; and interleaving the at leasttwo filtered data sets to produce the image.
 19. The method of claim 18,further comprising partitioning the amplitude function in the imageplane into two spatially distinct regions.
 20. The method of claim 19,further comprising selectively activating an electro-optically activematerial to apply the modulation pattern to one of the two spatiallydistinct regions in the image plane.